Transitivity and the Co-occurrence Relation in LSI

Current research in Latent Semantic Indexing (LSI) shows improvements in performance for a wide variety of Information Retrieval systems. Researchers use experimental methods to determine the appropriate number of dimensions for a given application. We propose the development of a theoretical foundation for determination of this parameter for LSI. We assert that LSI’s use of higher orders of co-occurrence is critical to this optimization function. In this work we present experiments that precisely determine the degree of transitivity used in LSI. We empirically demonstrate that LSI uses up to fourth order term co-occurrence. We also prove mathematically that a transitivity path exists for every nonzero element in the truncated term-term matrix computed by LSI. A complete understanding of the degree of transitivity will be key to understanding how a reflexive, symmetric and transitive relation based on the co-occurrence relation can form semantic equivalence classes for a collection.